Update: the Hoosier Audiovisual Multi-talker Database

Audiovisual speech recognition using multiscale nonlinear image decomposition

There has recently been increasing interest in the idea of enhancing speech recognition by the use of visual information derived from the face of the talker. This paper demonstrates the use of nonlinear image decomposition, in the form of a ‘sieve’, applied to the task of visual speech recognition. Information derived from the mouth region is used in visual and audiovisual speech recognition of...

Face or Voice? Determinant of Compellingness

This study examined sources of talker differences in the McGurk effect, by questioning which of auditory speech or visual speech is more determining the size of the McGurk effect. Cross-talker dubbing was done between faces and voices of utterances (/ba/ and /ga/) pronounced by two talkers: one compelling talker (CT) to the McGurk effect, and one less compelling talker (LT), as measured in our ...

Audiovisual benefit for recognition of speech presented with single-talker noise in older listeners

The effect of varying talker identity and listening conditions on gaze behavior during audiovisual speech perception.

During face-to-face conversation the face provides auditory and visual linguistic information, and also conveys information about the identity of the speaker. This study investigated behavioral strategies involved in gathering visual information while watching talking faces. The effects of varying talker identity and varying the intelligibility of speech (by adding acoustic noise) on gaze behav...

Mixing faces and voices: a study of the influence of faces and voices on audiovisual intelligibility

This study examined the influence of mixing faces and voices on the audiovisual intelligibility. The goal is to study the effect of combining two sources of information on the audiovisual intelligibility. Cross-talker dubbing was performed between faces and voices of 10 meaningful sentences pronounced by 10 talkers: 5 females and 5 males. Human subjects were asked to rate the articulation of th...